A0-A07*  941  UNIVERSITY  OF  SOUTHERN  CALIFORNIA  LOS  AN9ELES  SOCIAL  — CTe  f/%  ll/l 

RANK  WEI9HTIN4  IN  MULTIATTRIBUTE  UTILITY  DECISION  MAKIN4!  AVOID— ETC fU> 
SEP  7»  W  •  STILLWELL  »  N  COWARDS  N000U-79-C-003A 

UNCLASSIFIED  USC-991S99-ST  ^ 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  OF  STANDARDS- 1963- A 


1 


iH  (  P 

^1  g 

C5  '  ^ 

00  W 


— <  H 


3  3 


social  science 

research  institute 


RESEARCH  REPORT 

Rank  Weighting  in  Multiattribute 
Utility  Decision  Making  :  Avoiding 
The  Pitfalls  of  Equal  Weights 

William  G.  Stillwell 
Ward  Edwards 


Sponsored  By: 

Defense  Advanced  Research 
Projects  Agency 
Department  of  the  Navy 
Office  of  Naval  Research 


Monitored  By: 

Department  of  the  Navy 
Office  of  Naval  Research 
Pasadena  Branch  Office 
Contract  No.  N00014-79-C-0038 


Approved  for  Public  Release  ; 
Distribution  Unlimited  ; 
Reproduction  in  Whole  or  in  Part  is 
Permitted  for  Any  Use  of  the  U.S.  Government 


fll  Ml  ? 


September,  1979 


SSRI  RESEARCH  REPORT  79-2 


V 


80 


0  8  3 


Social  Science  Research  Institute 
University  of  Southern  California 
Los  Angeles,  California  90007 
(213)  741-6955 


INSTITUTE  GOALS: 

The  goals  of  the  Social  Science  Research  Institute  are  threefold: 

•  To  provide  an  environment  in  which  scientists  may  pursue  their 
own  interests  in  some  blend  of  basic  and  methodological  research 
in  the  investigation  of  major  social  problems. 

•  To  provide  an  environment  in  which  graduate  students  may 
receive  training  in  research  theory,  design  and  methodology 
througli  active  participation  with  senior  researchers  in  ongoing 
research  projects. 

•  To  disseminate  information  to  relevant  public  and  social  agencies 
in  order  to  provide  decision  makers  with  the  tools  and  ideas 
necessary  to  the  formulation  of  public  social  policy. 

HISTORY: 

The  Social  Science  Research  Institute,  University  of  Southern  California, 
was  established  in  1972,  with  a  staff  of  six.  In  fiscal  year  1978-79,  it  had 
a  staff  of  over  90  full-  and  part-time  researchers  and  support  personnel. 
SSRI  draws  upon  most  University  academic  Departments  and  Schools  to 
make  up  its  research  staff,  e.g.  Industrial  and  Systems  Engineering,  the 
Law  School,  Psychology,  Public  Administration,  Safety  and  Systems 
Management,  and  others.  Senior  researchers  have  joint  appointments  and 
most  actively  combine  research  with  teaching. 

FUNDING: 

SSRI  Reports  directly  to  the  Executive  Vice  President  of  USC.  It  is 
provided  with  modest  annual  basic  support  for  administration,  operations, 
and  program  development.  The  major  sources  of  funding  support  are 
federal,  state  and  local  funding  agencies  and  private  foundations  and 
organizations.  The  list  of  sponsors  has  recently  expanded  to  include 
governments  outside  the  United  States.  Total  funding  has  increased  from 
approximately  $150,000  in  1972  to  almost  $3,000,000  in  the  fiscal  year 
1978-1979. 

RESEARCH  INTERESTS: 

Each  senior  SSRI  scientist  is  encouraged  to  pursue  his  or  her  own  research 
interests,  subject  to  availability  of  funding.  These  interests  are  diverse: 
a  recent  count  identified  27.  Four  major  interests  persist  among  groups 
of  SSRI  researchers:  crime  control  and  criminal  justice,  methods  of 
dispute  resolution  and  alternatives  to  the  courts,  use  of  administration 
records  for  demographic  and  other  research  purposes,  and  exploitation  of 
applications  of  decision  analysis  to  public  decision  making  and  program 
evaluation.  Rut  many  SSRI  projects  do  not  fall  into  these  categories. 
Most  projects  combine  the  skills  of  several  scientists,  often  from  different 
disciplines.  As  SSRI  research  personnel  change,  its  interests  will  change 
also. 


— i 


RANKJVEIGHTING  IN  MULTIATTRIBUTE 

UTILITY  DECISION  MAKING:  AVOIDING 

<■  •-  • 

THE  PITFALLS  OF  EQUAL  WE  I  GITS 


Research  Report  79-2 


William  G.  Stillwell 


Ward/ Edwards 


Social  Science  Research  Institute 
University  of  Southern  California 


Sponsored  by 

Defense  Advanced  Research 
Projects  Agency 


'  it*- 


SUMMARY 


' 

The  notion  of  "dominance"  in  multiattribute  utility  decision 
contexts  leads  to  a  change  in  the  considered  alternative  set. 

The  impli cat ions  of  this  set  change  are  discussed  in  relation  to 
the  conditions  of  Wainer’s  (Wainer,  1976)  "equal  weights  theorem" 
and  the  resulting  sensitivity  to  weighting  of  importance  dimen¬ 
sions  demonstrated.  Data  from  three  multiattribute  decision  making 
studies  are  examined  using  four  rank  weighting  techniques  as  well 
as  equal  weights.  Rank  weighting  of  importance  dimensions  demon¬ 
strate  marked  improvement  of  approximation  as  reflected  in  both 
Pearson  and  rank  order  correlations  for  measures  of  overall 
utility  across  alternatives  within  the  nondominated  subset. 
Implications  for  multiattribute  utility  application  are  discussed. 
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1.  Introduction 

A  number  of  recent  articles  have  compared  equal  weighting  to 
various  statistical  weighting  schemes  for  linear  prediction  and  de¬ 
cision  models  (Dawes  and  Corrigan,  1974;  Newman,  1977;  Newman,  Seaver, 
and  Edwards,  1976).  Dawes  and  Corrigan's  (1974)  seminal  article  pro¬ 
vided  evidence  that  simplified  weighting  procedures  are  strikingly 
robust.  Both  equal  and  random  weighting  procedures  provided  excellent 
approximations  to  a  defined  optimal  set  of  weights  in  such  diverse 
settings  as  graduate  admissions,  psychiatric  diagnosis,  and  an  ab¬ 
stract  decision  task. 

Expanding  this  notion,  Einhom  and  Hogarth  (1975)  derived  the 
minimum  correlation  between  standard  linear  regression  and  unit  weight¬ 
ing  composites  as  a  measure  of  the  degree  of  similarity  of  those  com¬ 
posites  and  in  conclusion  stated  "The  minimum  is  high  for  most  applied 
situations."  They  contend  that  unit  weighting  is  a  viable  alternative 
to  standard  regression  methods  "because  unit  weights:  (1)  are  not  esti¬ 
mated  from  the  data  and  therefore  do  not  consume  degrees  of  freedom;  (2) 
are  estimated  without  error;  and  (3)  cannot  reverse  the  true  relative 
weights  of  the  variables."  In  addition,  of  course,  equal  weights  require 
no  elicitation. 

Wainer  (1976)  took  this  argument  one  step  further.  In  defense  of  his 
"Equal  weights  theorem"  Wainer  showed  that  under  "very  general  circum¬ 
stances"  equal  weights  can  replace  the  least  squares  weights  for  the  sam¬ 
ple  from  which  the  regression  weights  were  derived  with  little  or  no  loss 
in  accuracy.  He  also  showed  that  equal  weights  will  hold  up  better  than 
weights  derived  from  multiple  regression  when  new  samples  are  examined. 


Although  Wainer  made  some  errors  in  the  estimation  of  loss  of  ex¬ 
plained  variance  (Laughlin,  1978)  given  his  conditions,  the  conclusions 
reached  were  both  tenable  and  important.  In  fact,  in  many  real  prediction 
situations  equal  weights  provide  remarkably  good  approximations  to  dif¬ 
ferential  weights,  again  measured  by  correlations  between  predictions. 
Wainer 's  conditions  are  the  following,  (a)  All  predictor  variables  (at¬ 
tributes)  should  be  oriented  properly;  that  is,  for  all  of  them,  either 
more  should  be  better  than  less  or  less  should  be  better  than  more,  (b) 
Predictor  variables  (attributes)  should  be  positively  intercorrelated. 

The  former  condition  is  easy  to  ensure;  an  appropriate  rescaling 
will  always  solve  the  problem.  Even  peaked  (nonmonotonic)  functions  be¬ 
come  monotonic  when  the  nonmonotonic  measure  is  rescaled  into  utility 
or  some  other  monotonic  measure  of  desirability. 

Condition  B  is  the  heart  of  the  matter.  Considered  as  a  requirement 
for  prediction  --  which,  after  all,  is  what  correlations  are  intended  to 
do  --  it  is  not  a  problem.  GRE  scores  will  correlate  positively  with 
GPAs  over  a  population  of  applicants  to  most  graduate  departments.  The 
same  comment  can  be  made  about  many,  perhaps  most,  of  the  simple  descrip¬ 
tive  measures  that  are  typically  used  in  examples  by  proponents  of  equal 
weights. 

But  description,  by  itself,  is  not  usually  the  point  of  any  real- 
world  application  of  these  ideas.  Description  is  typically  intended  to 
be  useful.  A  description  can,  it  seems  to  us,  be  useful  only  if  it  serves 
as  a  basis  for  one  or  more  decision.  And  indeed  the  examples  used  by  the 
proponents  of  equal  weights  are  decision  examples:  admission  to  graduate 
school,  assignment  to  a  psychiatric  classification  (with  differential  im¬ 
plications  for  treatment),  and  so  on. 
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Once  we  conclude  that  decision,  rather  than  mere  description,  is  our 
goal,  a  set  of  considerations  quite  different  from  those  relevant  to  des¬ 
cription  alone  become  relevant  --  even  crucial.  This  paper  examines  some 
of  those  considerations  and  their  consequences;  we  are  looking  into  others. 
Some  of  the  ideas  of  the  riskless  part  of  multiattribute  utility  measure¬ 
ment  (MAUM)  are  very  similar  to  those  that  enter  into  the  equal -weights 
discussion  and  will  be  discussed  briefly  in  this  paper  as  they  apply. 

(For  expositions  of  various  approaches  to  and  applications  of  MAUM,  see 
Keeney  and  Raif fa,  (1976);  Edwards,  (1971;  1973;  1978). 

Newman,  in  work  done  earlier  but  published  in  1977,  showed  that  in 
at  least  some  multiattribute  decision  contexts  equal  weighting  of  attri¬ 
butes  could  lead  to  substantial  changes  inthe  resulting  evaluations. 
Following  up  Newman's  earlier  work,  Newman,  Seaver  and  Edwards  (1976), 
in  a  simulation  study,  found  that  equal  weighting  led  to  poor  selection 
ordering  in  most  situations  that  have  substantial  negative  correlations 
among  pairs  of  dimensions.  Keren  and  Newman  (1978)  demonstrated  addi¬ 
tionally  that  the  presence  of  suppressor  variables  (defined  by  Conger, 

1974)  among  dimensions  could  lead  to  marked  inferiority  of  equal  weighting 
to  regression  weighting,  even  for  pure  prediction  situations. 

The  key  factor  that  makes  decision  situations  different  from  pre¬ 
diction  situations  is  the  idea  of  irrelevance,  or  domination.  If  a  single 
option  is  to  be  chosen  in  a  multiattributed  contest,  then  only  options 
on  the  Pareto  frontier  are  admissible  candidates.  Dr.  David  Seaver,  while 
a  graduate  student  in  this  laboratory,  first  saw  the  point.  Unpublished 
work  he  performed  in  1973  (Note  1)  demonstrated  that  while  correlations 
between  aggregate  measures  of  overall  utility  were  insensitive  to  model 


changes  (additive,  multiplicative,  and  quasi -additive)  for  either  an 
entire  set  of  1000  options  or  the  nondominated  subset  of  100  options, 
within  the  nondominated  subset  these  correlations  were  extremely  sensi¬ 
tive  to  dimension  weights.  Specifically,  equal  weights  were  found  to 
produce  measures  of  overall  utility  which  markedly  differed  from  those 
produced  by  the  defined  set  of  optimal  weights. 

Table  1  gives  the  results  of  this  analysis.  Of  particular  interest 
are  the  correlations  of  weighting  procedure  1  (the  defined  optimal  weight 

set)  with  weighting  procedure  2  (differential  and  correctly  rank  ordered 
but  non-optimal)  and  weighting  procedure  1  with  weighting  procedure  3 
(equal  weights) .  The  explicit  implication  of  this  analysis  is  that  weights 
do  matter  when  nondominated  options  are  being  considered,  but  are  not 
crucial  as  long  as  they  are  correctly  rank  ordered  and  to  some  extent 
differential.  Implicit  in  the  analysis  is  that  negative  correlations 
exist  between  dimensions  within  the  nondominated  set  of  alternatives.  Al¬ 
though  his  paper  did  not  report  this  analysis,  Seaver  looked  at  the  cor¬ 
relations  between  dimensions  within  the  non-dominated  set  of  alternatives 
and  found  that  indeed  many  were  negative.  In  fact  9  out  of  a  possible  10 
of  these  correlations  for  the  5  dimensions  were  negative.  And  it  is  these 
negative  correlations  between  dimensions  that  are  responsible  for  the  poor 
performance  of  equal  weights. 

Following  up  on  the  insight  of  Seaver  and  the  implications  of  his 
study,  the  importance  of  which  is  only  now  beginning  to  be  understood. 


TABLE  1 
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Correlations  between  overall  utilities  for 
Different  models  and  weighting  schemes. 


all 

admissible 

alternatives 

alternatives 

alternatives 

1000 

100 

dimensions 

5 

5 

Q>  p 

.980 

.995 

•H  3  pl 

ir<  C£  r1 

.690 

.878 

O  p^ 

f  S  F3 

.825 

.867 

.660 

.504 

u  4->  4 

dimensions 

5 

5 

£  1x2 

.970 

.959 

•6  1x3 

.752 

.514 

S  1x4 

.335 

-.157 

£  1x5 

.164 

-.274 

2x3 

.870 

.692 

c  2x4 

.511 

.053 

'2]  2x5 

.324 

-.109 

m,  3x4 

.869 

.757 

'Z  3x5 

.743 

.630 

2  4x5 

.968 

.971 

1:  (81,  49,  25,  9,  1)  [optimal] 

2:  (9,  7,  5,  3,  1) 

3:  (1,  1,  1,  1,  1) 

4:  (1,  3,  5,  7,  9) 

5:  (1,  9,  25,  49,  81) 

models 

Fl(x)  -  E  x-  +  E  Ex.x. 

-1  jj)  1 

F2(x)  =  Exi+fxi 

F3(x)  +  (Xj^  +  X3-X4)-X5 

F4(x)  =  (Xj  +  X2)-X3  *  X4-X5 


.WE  from  "Corre¬ 
lation  Analyses  of 
Approximations  and 
Sensitivity  for  Multi- 
attribute  Utility." 

By  D.A.  Seaver,  unpub¬ 
lished  manuscript, 
Mathematical  Psychology 
Program,  University 
of  Michigan,  1973. 
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McClelland  (1978)  demonstrated  the  ubiquity  of  significant  negative  corre¬ 
lations  among  dimensions  in  multiattribute  utility  applications.  In  ad¬ 
dition  he  examined  the  problems  that  accrue  from  the  use  of  equal  weight¬ 
ing  of  dimensions  in  those  applications. 

The  argument  for  negative  correlations  among  dimensions  relevant 
to  decision  is  extremely  straightforward.  Good  sense  and  formal  argument 
alike  would  require  that,  in  any  decision  situation,  the  chosen  alterna¬ 
tive  should  not  be  clearly  inferior  to  some  other  alternative.  Option  B 
is  clearly  inferior  to,  or,  in  more  technical  language,  is  dominated  by 
option  A  if  A  is  at  least  as  good  as  B  on  every  relevant  dimension,  and 
definitely  better  on  at  least  one.  In  such  a  situation,  no  reasonable 
person  would  bother  to  consider  option  B;  he  would  simply  eliminate  it 
from  the  option  set.  The  Pareto  frontier  mentioned  above  is  simply  the 
highly  reduced  set  of  options  that  remain  for  consideration  after  domi¬ 
nated  options  are  eliminated. 

But  elimination  of  options  inevitably  changes  the  correlations 
among  all  pairs  of  dimensions.  Dimensions  that  may  have  been  positively 
correlated  in  the  original  option  set  almost  inevitably  become  negatively 
correlated  in  the  reduced  one.  This  means,  of  course,  that  within  the 
reduced  option  set  condition  B  of  W'ainer's  Equal  Weights  theorem  is  al¬ 
most  inevitably  violated. 

Consider  the  following  simple  example.  You  have  decided  to  buy  a 


new  car.  Your  options  are: 
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TABLE  2 

Alternative  Set 


Car  1 

10 

22 

24 

42 

Car  2 

37 

77 

99 

96 

Car  3 

89 

85 

28 

63 

Car  4 

9 

10 

7 

51 

Car  5 

2 

1 

52 

7 

Car  6 

36 

54 

32 

29 

Car  7 

2 

81 

29 

0 

Car  8 

5 

91 

0 

0 

Note: 

Entries 

for 

this  table 

randomly  generated 

Table 

3  shows 

the 

correlations  between  dimensions 

for 

the  eight  cars 

TABLE  3 


Attribute  correlations:  complete  choice  set 

A1  A2  A3  A4 

A1  1.0 

A2  .42  1.0 

A3  .20  .06  1.0 

A4  .57  .07  .57  1.0 

All  correlations  are  slightly  to  moderately  positive.  But  several 
cars  are  dominated.  For  instance,  car  1  has  values  (10,  22,  24,  42) 
while  car  2  has  values  (37,  77,  99,  96).  No  rational  person  would 
choose  car  1  when  he/she  could  choose  car  2,  no  matter  what  the 
weighting  of  the  attributes. 
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Similarly  car  3  dominates  car  4,  and  2  dominates  5  and  6.  Cars  2,  3,  7, 
and  8  are  the  remaining  options  upon  which  a  decision  analysis  would 
have  some  bearing.  Table  4  shows  the  intercorrelations  for  this  reduced 
choice  set.  Attribute  has  a  high  negative  correlation  with  both  A^ 
and  A^  in  violation  of  Wainer's  condition  B. 


TABLE  4 


Attribute 

correlations:  1 

reduced  choice  set 

A1 

A2 

A3 

A4 

A1 

1.0 

A2 

-.10 

1.0 

A3 

.20 

-.89 

1.0 

A4 

.67 

-.62 

.84 

1.0 

this  violation  mean 

?  In 

practice 

it  means 

weights  were  used  in  multiattribute  utility  measurement  of  either  additive 
or  multiplicative  types,  the  two  negatively  correlated  dimensions  would, 
to  some  extent,  cancel  each  other.  The  magnitude  of  the  cancellation 
effect  would  depend  on  the  size  of  the  correlation  and  on  the  weights 
of  the  dimensions.  In  the  car  example,  assume  an  individual's  true 
weights  to  be  the  following: 

A-, 


"1 

.4 


2 

.4 


4 

.1 


A  linear  multi  attribute  utility  analysis  of  the  alternative 

cars  would 

yield  these  results: 

Total  Utility 

Rank 

Car  2 

65.10 

2 

Car  3 

78.70 

1 

Car  7 

36.10 

4 

Car  8 

38.40 

3 

The  same  analysis  using  equal  weights 

produces  the  following 

conflicting 

results: 

Total  Utility 

Rank 

Car  2 

77.25 

1 

Car  3 

66.25 

2 

Car  7 

28.00 

3 

Car  8 

24.00 

4 

The  correlation  between  these  two  sets 

of  utilities  is  .90. 

But  equal 

weights  would  lead  to 

selection  of  car 

2  while  the  decision 

maker's 

true  weights  would  lead  to  the  selection  of  car  3.  The  potential  loss 
in  utility  calculated  from  the  decision  maker's  own  weights  is  a  strik¬ 
ing  17.3-o.  This  paper  presents  a  set  of  simplified  weighting  procedures 
that  avoid  the  problems  of  equal  weights  and  at  the  same  time  simplify 
the  assessment  task. 

II.  Rank  Weights 

Weighting  procedures  are  usually  compared  on  two  criteria,  ease 


of  elicitation  (for  both  subject  and  analyst)  and  numerical  quality. 

The  extremes  of  these  dimensions  are  represented  by  equal  and  ratio 
weights.  Equal  weights  present  no  elicitation  problem,  but  the  numbers 
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give  no  information  for  discrimination  between  non-domi nated  alternatives. 
Weights  that  have  appropriate  ratio  properties  are  the  most  difficult 
to  assess  by  any  procedure,  but  provide  the  maximum  discrimination  power. 
These  ratio  properties,  essential  to  multiattribute  utility  measurement, 
are  implicit  in  the  elicitation  procedures  reviewed  and  invented  in  the 
Keeney-Rai ffa  book  (1976),  and  are  explicitly  the  basis  of  Edwards' 
procedure  (Edwards,  1973;  1977;  Edwards,  Guttentag  and  Snapper,  1975). 

Newman  (1977)  and  Newman,  Seaver  and  Edwards  (1976)  discuss  equal  versus 
ratio  weights  thoroughly  but  leave  unanswered  the  question:  How  simple 
can  elicitation  be  and  still  give  adequate  representation  for  a  good 
decision? 

Obviously  equal  weights  are  the  simplest  to  assess,  and  in  a  number 
of  prediction  contexts  they  have  been  shown  to  be  of  sufficient  quality 
to  justify  their  use  (Newman,  Seaver,  and  Edwards,  1976;  Dawes  and  Corrigan, 
1974).  The  concern  of  this  paper  is  those  situations  in  which  some  level 
of  differential  weighting  is  necessary,  yet  the  difficulty  of  eliciting 
ratio  weights  demands  some  intermediate  procedure.  Rank  weighting  provides 
a  simplified  assessment  procedure  and  a  method  of  assigning  weights  that 
does  not  have  the  limitations  of  equal  weighting. 

This  paper  examines  four  rank  weighting  procedures.  All  four  fall 
between  equal  and  ratio  weights  both  in  assessment  ease  and  in  number 
quality.  Each  derives  information  from  the  rank  ordering  of  the  attributes 
in  the  attribute  set.  This  information  determines  the  weight  given  the 
attribute  in  the  subsequent  MAUM.  We  have  named  the  weighting  techniques 
rank  sum,  rank  reciprocal,  decision  rule  rank  and  rank  exponent. 


?*$**#$$* 


Rank  sum  weights  are  the  standard  rank  weighting  technique 
considered  in  the  literature.  N  attributes  are  ranked  and  each  attri- 
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bute  is  weighted  (N-R.+l)  where  R.  is  the  rank  position  of  the 

N 

attribute.  Each  weight  is  then  normalized  by  I  (N-R.  +1)  so  that 

i=l  1 

the  weights  sum  to  1.0.  Table  5  shows  rank  sum  weights  as  well  as  rank 
reciprocal  and  rank  exponent  weights  for  5  dimensions.  Equal  weights 
would,  of  course,  assign  a  weight  of  .2  to  each  dimension.  The  problem 


of  tied  ranks  among  dimensions  will  be  dicussed  later. 

TABLE  5 

Rank  weights  for  5  dimensions 


Dimension 

Rank  Sum 

Rank  Reciprocal 

Rank  Exponent 

,  Rank 

Weight 

Weignt 

Weight  (Z=l .5) 

1 

.333 

.438 

.396 

\  2 

.267 

.219 

.284 

3 

.200 

.146 

.184 

4 

.067 

.088 

.035 

For  rank  reciprocal  weights,  the  reciprocal  of  an  attribute's  rank 
is  taken  as  that  attribute's  weight.  That  is,  the  weights  before  normali¬ 
zation  are  1,  1/2,  1/3,  1/4,  and  1/5.  Each  weight  is  then  normalized, 
as  in  other  rank  weight  procedures.  Decision  rule  rank  weights  require 
one  more  piece  of  information  from  the  decision  maker.  Once  he  has 
rank  ordered  the  attributes,  he  must  judge  the  weight  of  the  most  important 
dimension.  Then  equal,  rank  sum,  or  rank  reciprocal  weights  are  used;  the 
method  of  choice  is  that  which  most  closely  approximates  the  weight 
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Rank  exponent  weights  require  the  same  information  from  the 
subject  as  decision  rule  rank  weights.  The  weight  given  the  most 
important  dimension  (W)  is  entered  into  the  following  formula: 


N  7 

z  (n  -  r1  +  ir  (i) 

i=l 

which  is  then  solved  for  Z  via  an  iterative  process.  N  is  the  number 
of  dimensions  in  the  ranking  and  is  the  rank  of  the  ith  dimension. 
Once  Z  is  known,  weights  for  the  rest  of  the  dimensions  are  determined 
For  instance  the  weight  for  the  dimension  ranked  third  would  be: 


l  (N  -  R,  +  1) 

1*1  1 

Table  5  shows  rank  exponent  weights  calculated  for  5  dimensions  using 
an  arbitrary  Z  value  of  1.5. 

The  first  step  in  any  of  these  methods  is  to  delete  from  the  list 
of  dimensions  any  dimension  to  which  the  respondent  wishes  to  assign 
zero  weight.  Ties  are  permitted  in  all  four  methods.  For  instance, 
the  ranks  in  a  five-dimensional  problem  might  be  5,  4,  2.5,  2.5,  and  1 
Each  of  these  methods  is  easy  to  apply  to  hierarchical  multi¬ 
attribute  utility  structures  (Value  Trees).  Each  branch  of  the  tree 
is  weighted  separately.  We  have  seen  no  published  discussion  of  the 
rather  trivial  arithmetic  by  which  such  separately  obtained  branch 
weights  can  be  aggregated  into  final  weights,  combined  with  single¬ 
dimension  utilities,  and  thereafter  be  appropriately  disaggregated 
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to  produce  useful  profiles  showing  how  each  option  stands  on  each  major 
value  within  the  tree.  Such  a  discussion,  with  a  real  world  example, 
appears  in  Edwards  (Note  2). 

Rank  exponent  weights  exhibit  several  interesting  characteristics. 

Z=0  defines  the  equal  weights  case  and  Z  =  1  defines  rank  sum  weights. 

As  Z  increases,  the  set  of  normalized  weights  gets  steeper  and  steeper. 

One  great  advantage  of  the  procedure,  considered  as  an  approximation, 
is  that  it  always  gives  exactly  the  same  weight  to  the  most  important 
dimension  that  the  decision  maker  does.  This  ensures  that  the  procedure 
will  yield  a  highly  satisfactory  approximation  to  the  ultimate  utilities 
from  a  multiattribute  utility  analysis. 

III.  Three  Examples 

In  this  section  we  reanalyze  the  data  from  two  decision  problems 
to  which  MAUM  has  been  applied.  In  addition  we  briefly  review  a  third 
reanalysis,  performed  by  Newman  in  this  laboratory.  Each  raises  an 
interesting  issues  about  the  use  of  rank  weighting  procedures.  For  detailed 
reports  of  the  three  problems  see  Edwards  (Note  2),  where  MAUM  was 
used  to  evaluate  the  desegregation  plans  for  the  Los  Angeles  Unified 
School  District,  Otway  and  Edwards  (1977),  a  study  of  siting  decisions 
for  nuclear  waste  disposal,  and  Newman  (1976)  where  automobile  selection 
was  at  issue. 

III.  A.  Desegregation 

In  July,  1977,  the  Los  Angeles  Unified  School  District  asked 
Edwards  to  design  a  method  of  analysis  of  various  desegregation  plans 
submitted  to  the  Los  Angeles  School  Board  as  a  near-final  step  in  the 
prolonged  Crawford  desegregation  case. 
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Seven  plans  were  chosen  for  evaluation.  Unfortunately,  the  plan  finally 
adopted  by  the  board  could  not  be  among  them,  since  it  was  prepared  too 
late  for  evaluation. 

Prolonged  discussions  with  senior  District  officers,  School  Board 
members,  plaintiffs  and  intervenors  in  the  Crawford  case,  and  others  led 
to  a  very  complex  Value  Tree.  The  tree  had  four  levels  and  a  total  of 
144  twigs.  (A  twig  is  the  lowest  level  of  a  downward-branching  tree; 
measures  of  performance,  i.e.  single-dimension  utilities,  must  be  collected 
for  each  twig).  The  techniques  Edwards  used  to  elicit  value  dimension, 
importance  weights,  and  location  measures  (single  dimension  utilities), 
were  those  of  SMART  (Edwards  1973;  1977),  suitably  modified  to  meet  the 
size  and  political  requirements  of  the  problem  and  the  lack  of 
decision- theoretical  expertise  of  the  numerous  respondents.  The  most 
important  politically  necessary  compromise  was  the  use  of  so  complex  a 
tree.  144  twigs  is  far  too  many!  But  none  could  be  eliminated  without 
offending  one  or  another  of  the  very  numerous  stakeholders. 

Eight  individuals  and  one  group  judged  weights,  using  the  standard 
SMART  ratio  judgment  technique.  Of  the  eight  individuals,  five  were 
members  of  the  Board;  their  weights  were  averaged  to  preserve  individual 
anonymity.  The  other  three  were  desegregation  experts. 

The  hierarchical  structure  of  the  Value  Tree  made  such  judgments 
possible.  Judgments  were  made  separately  for  each  branch;  the  smallest 
branch  had  only  two  values  and  the  largest  had  14.  Several  techniques 
were  used  to  cut  down  excessive  judgmental  labor  in  the  lower  branches 


of  the  tree. 


We  reanalyze  only  the  averaged  weights  of  the  five  Board  members 
here.  Each  of  the  four  rank  weighting  techniques,  as  well  as  equal 
weights,  were  used  to  calculate  the  overall  value  of  each  plan.  In  the 
case  of  rank  reciprocal  and  rank  sum  weights,  the  number  of  dimensions 
determined  the  weight  given  to  each  dimension  in  the  analysis.  For 
decision  rule  rank  and  rank  exponent  weights,  the  weight  given  the  most 
important  dimension  for  averaged  ratio  weights  was  used  to  determine  the 
complete  set  of  individual  dimension  weights. 

The  entire  analysis  was  conducted  for  each  set  of  weights.  Raw 
data  for  plan  values  is,  of  course,  that  used  in  the  original  analysis. 
The  plans  were  examined  for  admissibility  and  none  found  to  be  dominated 
Values  of  overall  benefit  were  calculated  for  each  weighting  technique 
for  both  the  1978-79  and  1981-82  fiscal  years.  Overall  benefit  shows 
strong  correspondence  between  results  for  each  of  the  rank  weighting 
techniques  as  well  as  equal  weights.  To  look  at  the  exact  strength  of 
this  relationship  we  calculated  Pearson  correlation  coefficients, 
correlating  aggregate  benefits  of  each  of  the  seven  plans  as  evaluated 
by  each  approximation  and  with  actual  Board  weights.  No  correlation 
not  involving  equal  weights  was  less  than  .98;  no  correlation  involving 
equal  weights  was  higher  than  .92. 

For  decision  making  purposes  a  second  criterion  is  change  in  the 
rank  ordering  of  alternatives  by  the  various  techniques.  To  examine 
this  criterion  plans  were  rank  ordered  by  each  weighting  technique  and 
a  Kendall  tau  coefficient  calculated.  A  Kendall  tau  coefficient  repre¬ 
sents  a  difference  between  two  proportions;  the  proportion  of  pairs 


of  measurements  having  the  same  relative  order  in  a  pair  of  rankings 
minus  the  proportion  of  pairs  showing  different  relative  order  in  two 
rankings  (Hays,  1973).  Thus  it  gives  us  a  measure  of  the  proportion 
of  changes  in  the  rank  ordering  in  a  pair  of  rank  ordered  sets.  Table 
6  shows  the  results  of  this  analysis  for  the  first  and  second  levels 
of  the  MAUM  structure. 

To  avoid  over-interpreting  Table  6  it  is  important  to  know  that 
all  weighting  procedures  led  to  the  same  plan  being  identified  as  best. 
Still  we  hope  to  be  able  to  generalize  to  other  decision  problems  where 
a  k  out  of  N  selection  rather  than  a  1  out  of  N  is  to  be  made.  For  the 
most  general  case  the  rank  ordering  of  all  alternatives  is  important. 

If  7  graduate  applicants  are  to  be  chosen  out  of  8  who  meet  the  minimum 
standards  of  selection,  reversal  of  the  rank  ordering  of  the  7th  and 
8th  applicants  would  lead  to  a  suboptimal  decision  while  reversal  of 
the  1st  and  2nd  ranked  applicants  would  make  no  difference. 

The  rank  order  correlations  in  Table  6  demonstrate  the  clear 
superiority  of  all  rank  weighting  techniques  to  equal  weights.  For  the 
1978-79  data  a  loss  in  tau  for  overall  utility  of  23. 8«  (.929-. 691  ) 
results  from  the  use  of  equal  weights  rather  than  the  next  poorest 
simpl if ir  >tion.  The  resultant  loss  for  1981-82  is  31.1*  (.929-. 618). 
Thus,  a  much  larger  proportion  of  rank  reversals  occur  with  equal  weights 
than  with  any  of  the  rank  weighting  techniques. 
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TABLE  6 

Kendall  tau  coefficients  between  plan  values  based  on  ratio  weights 
and  plan  values  produced  by  approximations. 

1978-79 


Equal 

Rank  Sum 

Rank  Reciprocal 

Decision 
Rule  Rank 

Rank 

Exponent 

Overal 1 
Uti 1 ity 

.691 

.982 

.929 

1.000 

1.000 

Branch  A 

1 .000 

.982 

.964 

.982 

1.000 

Branch  B 

.890 

.944 

.815 

1.000 

1.000 

Branch  C 

.929 

.982 

.929 

1.000 

.982 

Branch  D 

.643 

.982 

.909 

1.000 

.982 

Branch  E 

.857 

.857 

.929 

.857 

.909 

Branch  F 

.905 

1 .000 

1.000 

1.000 

1.000 

1981-82 

Overal 1 
Uti 1 ity 

.618 

.929 

.929 

1 .000 

.982 

Branch  A 

1 .000 

.929 

.929 

1 .000 

1.000 

Branch  B 

.982 

.982 

.963 

.963 

.982 

Branch  C 

.764 

.982 

.889 

1.000 

.963 

Branch  D 

.714 

.982 

.982 

.982 

.982 

Branch  E 

.837 

.889 

.909 

.889 

.909 

Branch  F 

.804 

.972 

1.000 

.972 

1.000 
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An  additional  test  of  weighting  approximation  is  to  compare  sets 
of  weights  directly.  The  primary  difference  among  the  various  approxima¬ 
tions  is  the  peakedness  of  the  weight  set.  Equal  weights,  of  course, 
have  no  peak.  Rank  sum  weights  are  less  peaked  than  rank  reciprocal 
weights.  Decision  rule  rank  and  rank  exponent  weights  are  determined 
by  the  judged  weight  of  the  most  important  dimension.  Only  the  rank 
exponent  procedure  (among  those  considered  here)  can  produce  weights 
more  peaked  than  rank  reciprocal. 

In  order  to  examine  the  relative  peakness  of  the  approximation 
techniques  relative  to  the  judged  ratio  weights,  regression  analyses 
were  performed  using  the  ratio  weights  as  predictors  of  weights  from 
each  approximation  technique.  Slopes  of  the  regression  lines  indicate 
the  peakedness  of  the  weights  of  the  approximation  technique  relative 
to  ratio  weights.  A  weight  set  that  perfectly  approximates  the  ratio 
weights  would  result  in  a  slope  of  1.0.  Table  7  shows  the  results  of 
this  analysis.  This  analysis  was  conducted  for  the  highest  level  of 


the  tree  as  well  as  each  second  level  set  of  weights.  Equal  weights  have 
no  variance  and  therefore  the  slope  would  equal  0.0. 

Intercepts  for  the  analyses  were  all  virtually  zero  and  thus  are 
not  included  in  this  table.  For  the  three  comparisons  that  are  possible 
rank  exponent  weights  always  produce  slopes  that  rank  first  or  second 
in  terms  of  absolute  distance  from  1.0.  This  is  not  surprising.  Rank 
exponent  weights  guarantee  that  the  weight  to  the  most  important  dimension 
will  be  correct  and  thus  the  regression  analysis  will  have  an  upper  end 
point  that  is  anchored  to  the  correct  ratio  weight.  It  is  interesting 
to  note  that  in  each  comparison  between  rank  reciprocal  and  rank  sum 
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Slopes  based 

TABLE  7 

on  regression  of  approximate  weights  on  ratio  weights. 

Rank  Sum 

Rank  Reciprocal 

Rank  Exponent 

1st  Level 

.641 

.955* 

1.101 

2nd  Level 

A  Branch 

1.551 

2.279 

1.005 

B  Branch 

1 .078* 

1.815 

1.159 

C  Branch 

.861* 

1.214 

.996 

D  Branch 

1.136* 

1.244 

.944 

E  Branch 

1 .002* 

1.367 

1 .165 

F  Branch 

.806* 

1.376 

1.208 

♦Weighting  procedure  selected  by 

the  decision  rule 

rank  procedure. 

(For  branch 

A  the  decision  rule 

rank  procedure  led 

to  the  selection 

of  equal  weights . ) 


weights,  the  decision  rule  procedure  would  have  chosen  the  weighting 
technique  whose  slope  was  closest  to  1.0.  In  addition,  for  the  case 
in  which  the  decision  rule  procedure  indicated  equal  weights,  both 
rank  sum  and  rank  reciprocal  weights  were  much  too  peaked  (indicated 
by  slopes  much  higher  than  1.0). 

III.  B.  Nucl ear  waste  disposal  site  selection 

Otway  and  Edwards  (1977)  presents  Multi-Attribute  Utility  Measure¬ 
ment  (MAUM)  as  a  method  of  evaluating  proposed  nuclear  waste  disposal 
sites.  The  paper  demonstrates  the  usefulness  of  MAUM  in  combining 
technical  information  with  the  corresponding  social  attitudes.  Although 
aware  of  the  importance  of  these  atti tudes .technical  experts  typically 
have  been  unable  either  to  measure  them  or  to  aggregate  them  with 
technical  data.  In  the  past,  either  or  both  of  these  problems  have 
led  the  experts  to  make  recommendations  that  overweight  the  importance 
of  the  engineering  aspects  of  the  problem.  Of  special  interest  is  the 
recent  development  of  methodologies  for  the  measurement  of  social 
attitude  toward  technologies  (Otway  and  Fishbein,  1976),  hopefully 
solving  the  measurement  problem.  MAUM,  of  course,  provides  a  means 
of  aggregation. 

The  experts  whose  information  was  used  for  the  analysis  were  ten 
high  ranking  specialists  from  eight  countries  with  advanced  nuclear 
energy  programs.  They  were  assembled  for  an  i nternational  meeting 
concerned  with  nuclear  waste  disposal.  The  experts  were  very  much 
concerned  with  the  problems  of  disposal  and  the  risks  involved  and 
were  thus  highly  co-operative. 


TABLE  8 
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Description  of  Six  Hypothetical  Nuclear  Waste  Disposal  Sites 


Value  Dimension,  Range  and 
Seal i ng 

Site  1  Site  2  Site  3  Site  4  Site  5  Site  6 


D1 .  Public  attitude.  0  = 

extremely  negative;  40  20  10  40  60  70 

100  =  extremely  positive 

D2.  Remoteness  from  popu¬ 
lation  center,  km. 

(0  km  =  0;  160  km 
-  100 

D3.  Geospheric  path  length, 
km  (0  km  -  0; 

160  km  =  100) 

D4.  Proximity  to  natural 
resources,  km.  (0  km  = 

0;  160  km  =  100) 

D5.  Geologic  disturbance 
probability  per  year 
(1  =  0;  10'6  =  100; 

1 i near  i n  exponent) 

D6.  Relative  migration  rate 

of  critical  nuclide  10"^  10'^  10"4  10”'  10"^  10 

(1  =  0;  10'5  =  100; 
linear  in  exponent) 

D7.  Transportation  distance 

km.  (1600  km  =  0;  1500  500  500  1500  150  150 

0  Lit.  --  100) 


40  12  12 

40  12  12 

50  150  50 

10-4  ] q-5  10’4 


120  40  120 

4  4  40 

50  15  15 

l0-6  10~b  10'G 


Note:  From  "Application  of  a  simple  multiattribute  rating  technique  to 
evaluation  of  nuclear  waste  disposal  sites:  A  demonstration."  by 
H.J.  Otway  and  W.  Edwards,  International  Institute  for  Applied  Systems 
Analysis  Research  Memorandum,  1977. 
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For  the  purposes  of  the  analysis  six  hypothetical  sites  were  invented. 
One  of  the  experts  at  the  conference  volunteered  to  perform  this  task. 

He  did  so  by  thinking  of  real  sites  that,  in  the  pasts,  had  been  suggested 
for  the  purpose.  The  relevant  physical  parameters  were  approximate 
figures.  Public  attitudes  toward  the  hypothetical  waste  disposal 
sites  were  assigned  on  a  random  basis.  Table  8  provides  a  numerical 
description  of  sites. 


As  in  the  previous  problem  the  SMART  procedure  was  used  in  each 
stage  of  the  analysis.  The  experts  described  above  provided  the  inputs. 

In  addition,  the  authors  provided  suggestions  for  dimensions,  which  the 
experts  accepted.  These  suggestions  dealt  primarily  with  public  attitude. 
Scaling  of  the  location  measures  was  also  provided  by  the  authors,  but 
in  conjunction  with  the  technical  experts. 

The  final  decision  structure  was  not  hierarchical.  Each  proposed 
waste  disposal  site  was  evaluated  on  seven  dimensions.  They  were: 
public  attitude  toward  the  site  remoteness  form  population  center, 
distance  from  the  nearest  point  used  by  the  public  (called  geospheric 
path  length  by  these  nuclear  specialists) , proximity  to  natural  resources, 
geologic  disturbance  probability,  relative  migration  rate  of  critical 
nuclide,  and  the  transportation  distance  between  the  site  and  the  nuclear 
plant  it  serves.  All  other  dimensions  were  assumed  equal  for  all  sites 
(for  a  discussion  of  nuclear  waste  disposal  see  Burkholder,  1976). 
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Importance  weights  of  the  value  dimensions  were  judged  twice.  Thus, 
test-retest  reliability  of  these  judgments  could  be  calculated.  The 
authors  perform  these  calculations  and  report  a  mean  correlation  of  .93 
with  a  standard  deviation  of  .11.  The  lowest  is  .65.  The  second  set 
of  weights  was  used  in  the  original  analysis  and  will  be  used  for  the 
analyses  in  this  paper.  Correlations  were  also  calculated  between  weights 
judged  by  all  possible  pairs  of  respondents  for  the  second  set  of  judgments. 
These  were  reported  to  range  from  .97  to  -.27  with  a  mean  of  .39  and 
standard  deviation  of. 35.  Normalized  averaged  weights  were  then  calculated 
across  respondents  and  these  means  used  in  the  utility  analysis. 

In  order  to  examine  the  strength  of  Wainer's  assertion  that  high 
negative  correlations  between  dimensions  occur  rarely  (Wainer,  1976), 
correlations  were  calculated  for  the  values  in  this  analysis.  In  seeming 
conflict  with  Wainer,  some  extremely  high  negative  correlations  occur. 

Over  half  (11  out  of  21)  interdimensional  correlations  are  negative 
with  several  less  then  -.6  (see  table  4,  Otway  and  Edwards,  1977). 

Obviously  some  serious  questions  can  be  raised  about  the  applicability 
of  Wainer's  result,  at  least  for  MAUM  decision  making  contexts. 

As  originally  scaled,  the  alternatives  to  be  evaluated  did  not 
cover  the  full  range  of  value  on  some  attributes (see  table  8).  This 
is  most  striking  for  the  third  dimension,  geospheric  path  length,  where 
the  range  of  the  alternatives  on  that  attribute  covers  only  22.5  of 
the  range  assigned  to  it.  When  a  set  of  alternatives  cover  only  a 
portion  of  the  total  range  of  an  attribute  the  importance  of  that 
attribute  is  diminished  proportionately  (  for  linear  utilities)  to  the 
part  of  the  range  not  covered.  That  decrease  is  not  represented  in  the 
original  importance  weight. 


This  problem  can  only  be  fully  solved  by  judgmental  methods;  that 


is,  reassessment  of  weights  with  attention  to  the  range  of  values  on 
alternatives.  While  a  reassessment  was  not  possible  for  this  analysis, 
a  transformation  procedure  was  undertaken  on  the  original  weights  in 
order  to  place  all  of  the  scaling  value  on  the  importance  weighting. 

This  placement  is,  of  course,  required  for  the  use  of  rank  weights.  The 
transformation  was  as  follows: 


100(uij-Mi)/Ri 


W'  =  W^/S 


where  S  =  '~W  ■  R  ■  . 

i  1  1 

R.  is  the  range  of  u^j  in  dimension  i  over  the  set  of  entities  to  he 
evaluated,  and  M.  is  the  minimum  value  of  u—  over  those  entities  in 
dimension  i.  So  u1  —  will  have  a  minimum  value  of  0  and  a  maximum  o 
100  on  each  dimension,  over  the  set  of  entities  to  be  evaluated. 

Table  9  shows  the  transformed  weights  as  well  as  the  rank  weight 
from  each  of  the  three  rank  weighting  procedures.  Equal  weights  would 
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give  a  weight  of  .143  to  each  of  the  seven  dimensions.  The  decision 
rule  weight  selection  procedure  would  lead  to  the  use  of  rank  sum  weights 
This  constitutes  all  the  information  necessary  for  the  analysis. 

Each  alternative  was  evaluated  using  each  set  of  rank  weights  as  well 
as  equal  weights  and  the  original  ratio  weights  elicited  from  the  experts 
As  in  the  LASD  analysis,  Pearscn  correlations  were  calculated  on  values 
of  overall  utility  between  weighting  techniques  across  potential  sites. 
Overall  utility  calculated  from  ratio  weighting  correlated  .86,  .99, 

. 72,  and  .97  with  overall  utility  calculated  from  equal,  rank  sum, 
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TABLE  9 


Weights  elicited  from  experts  and 


those  from  rank  weighting  procedures 


for  Otway  and  Edwards  (1977). 


Value  dimension 

Transformed 

Ratio 

Weights 

Rank  Sum* 
Weights 

Rank  Reciprocal 
Weights 

Rank  Exponent 
Weights  z-  1. 

Public  Attitude 

.246 

.214 

.193 

.223 

Distance  from 

City 

.087 

.071 

.064 

.062 

Geospheric  Path 
Length 

.088 

.107 

.077  . 

.100 

Proximity  to 
Natural  Resources 

.267 

.250 

.386 

.267 

Earthquake  Probability. 103 

.143 

.096 

.139 

Migration  of 
Critical  Nuclide 

.153 

.179 

.129 

.181 

Transportation 

Distance 

.055 

.036 

.055 

.028 

62 


♦Decision  Rule  Rank  Weights  would  be  the  same  as  Rank  Sum  Weights 
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rank  reciprocal  and  rank  exponent  weights  respectively.  For  this 
analysis  both  rank  sum  and  rank  exponent  are  markedly  superior  to  equal 
weights.  It  is  especially  interesting  to  note  that  the  decision  rule 
procedure  led  to  the  selection  of  rank  sum  weights,  clearly  the  optimal 
choice  from  among  the  approximation  procedures. 

Kendall  tau  coefficients  were  calculated  as  in  the  previous 
analysis.  Tau  was  .733,  .867,  .333  and  .733  between  overall  utilities 
calculated  from  ratio  weights  and  those  calculated  from  equal,  rank  sum, 
rank  reciprocal  and  rank  exponent  weights  respectively.  Slopes  were  cal¬ 
culated  between  ratio  weights  and  each  approximation  weight  set.  Slopes 
were  .874,  1.27,  and  .990  between  ratio  weights  and  rank  sum,  rank  reci¬ 
procal  and  rank  exponent  weights  respectively. 

It  seems  apparent  from  a  comparison  of  the  correlations  and  slopes 
that  while  rank  sum  weights  prove  superior  for  the  alternatives  in  this 
problem,  it  is  quite  likely  that  rank  exponent  weight  would  prove 
superior  generally.  That  is,  for  this  analysis  rank  sum  weighting 
resulted  in  higher  correlations  between  measures  of  overall  utility 
but  rank  exponent  weights  have  slopes  much  closer  to  1.0  and  therefore 
for  a  different  set  of  alternatives  would  prove  the  superior  procedure. 
III.  C.  Automobile  Selection. 

A  third  brief  example  comes  from  work  performed  by  J.R.  Newman  in 
his  reanalysis  of  a  study  of  optimal  automobile  design.  The  Automobile 
Club  of  Southern  California  developed  a  target  program  in  which  it  sought 
to  evaluate  the  contribution  of  various  factors  or  attributes  to  an 
automobile's  overall  value  to  a  cross  section  of  the  driving  public. 

The  club's  members  and  other  experts  provided  the  list  of  attributes. 
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Eleven  attributes  were  identified  consisting  of  fuel  economy,  interior 
size,  passing/acceleration  ability,  interior  noise,  exterior  size, 
crashworthiness,  luggage  capacity,  handling,  ride  quality,  ease  of  entry 
and  exit  and  maneuverability..  Expert  staff  of  the  Auto  Club  itself 
provided  location  measures  on  each  dimension.  31  non-domi nated  1978 
automobiles  were  used  as  alternatives  for  the  anlaysis. 

Interdimensional  correlations  were  calculated.  Correlations 
between  attributes  cover  almost  the  full  range  from  -1  to  +1.  Almost 
half  (27  of  55)  of  the  correlations  are  negative  with  6  lower  than  -.6. 

As  previously  stated,  these  high  negative  correlations  imply  severe 
problems  when  equal  weights  are  used  for  the  analysis.  This  implied 
problem  is  illustrated  by  the  comparison  of  the  results  from  the  various 
weighting  schemes.  Again  Pearson  correlations  were  calculated  between 
weighting  techniques,  across  alternatives.  The  correlations  of  ratio 
weighting  with  equal,  rank  sum,  rank  reciprocal,  and  rank  exponent  are 
.726,  .800,  .925,  and  .820  respectively.  Equal  weights  clearly  provide 
the  worst  approximation  to  the  ratio  weights.  Rank  sum  and  rank  exponent 
are  similar  and  rank  reciprocal  is  by  far  the  best  approximation  to 
ratio  weights.  As  in  the  nuclear  disposal  site  selection  reanalysis 
the  decision  rule  selection  procedure  results  in  a  choice  of  the  weighting 
technique  that  has  the  highest  correlation  with  ratio  weights. 

Kendall  tau  coefficients  between  overall  utilities  were  calculated 
along  with  slopes  between  weight  sets.  Tau  was  .588,  .632,  .788,  and  .600 
between  the  results  from  ratio  weights  and  those  from  equal,  rank  sum,  rank 
reciprocal  and  rank  exponent  weights  respectively.  Slopes  between  ratio 
and  rank  sum,  rank  reciprocal  and  rank  exponent  were  .685,  1.397  and  1.231. 
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The  information  provided  by  these  two  indices  seems  to  be  in  conflict. 

While  the  slopes  suggest  that  for  the  more  general  situation  rank  exponent 
weights  would  provide  the  best  approximation,  the  correlations  between 
measures  of  overall  utility  point  to  rank  reciprocal  (for  this  set  of  al¬ 
ternatives  at  least).  Still,  all  of  the  rank  weight  procedures  result  in 
higher  correlation  than  do  equal  weights,  lending  evidence  for  the  use  of 
some  rank  weighting  procedure. 

Discussion 

The  use  of  rank  weighting  techniques  as  approximations  to  ratio 
weights  in  this  study  provided  remarkably  good  results.  In  each  of  the 
three  reanalyses  at  least  one  of  the  rank  weighting  approximations 
resulted  in  a  Pearson  correlation  coefficient  between  ratio  and  rank 
weights  for  aggregate  overall  utilities  above  .9.  Also,  in  each  of  the 
re-analyses  equal  weighting  of  attributes  led  to  a  sizable  loss  in 
correlation  for  measures  of  overall  value.  Thus,  rank  weighting  of 
attributes  can  be  said  to  provide  good  approximations  to  the  "true" 
ratio  weights.  Within  the  conditions  of  this  study  and  for  the 
purposes  of  multiattribute  decision  making,  rank  weights  seem  to  be 
sufficient  improvements  over  equal  weights  to  warrant  the  extra  effort 
involved  in  their  elicitation. 

Our  ability  to  validate  an  approximation  technique  is  obviously 
bounded  by  the  quality  of  the  judgment  or  value  that  is  being  approxi¬ 
mated.  Because  this  study  depends  so  heavily  on  the  ratio  weights  elicited 
from  the  subjects,  some  rationale  should  be  developed  for  their  use  as 
a  criterion.  The  use  of  ratio  weights  as  a  criterion  rests  on  at  least 
two  assumptions.  First,  we  are  assuming  that  an  individual  knows  his/her 
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own  mind;  that  is,  that  he/she  is  able  to  determine  the  rank  ordering  of 
dimension  importance,  the  relative  strengths  of  the  dimension  weighting 
and  a  true  zero  point.  Secondly,  we  are  assuming  that  an  individual 
can  express  this  information  in  quantitative,  orderly  fashion. 

Support  for  both  of  these  assumptions  comes  from  three  studies. 

Each  study  examines  weighting  in  the  context  of  the  multiple  cue 
learning  paradigm  (Hammond  and  Summers,  1972).  Cook  and  Stewart  (1975), 
in  a  task  where  subjects  were  asked  to  evaluate  graduate  student 
profiles,  found  that:  1)  various  methods  for  elicitation  of  subjective 
weights  (the  weights  that  the  subjects  say  they  are  using)  result  in 
roughly  equivalent  judgments,  i.e.,  consensual  validation,  and  2)  these 
subjective  weights  correspond  fairly  closely  to  objective  weights 
derived  from  the  best  fit  linear  model  (i.e.,  linear  regression).  Both 
of  these  conclusions  suggest  that  subjects  can  express  their  decision 
policies  in  an  orderly  quantitative  fashion. 

Two  other  studies  which  lend  support  to  the  notion  that  subjects 
can  accurately  describe  their  decision  policies  are  those  of  Schmitt 
(1978)  and  John  and  Edwards  (Note  3).  In  each  of  these  studies  subjects 
were  trained  in  a  weighting  model  using  the  multiple  cue  learning 
paradigm  (Hammond  and  Summers,  1972)  and  then  asked  to  describe  their 
subjective  judgment  as  to  the  weighting  of  dimensions.  Each  study 
used  a  number  of  elicitation  techniques  for  the  report  of  these  weights. 

In  both  of  these  studies  the  method  of  eliciting  the  subjective  weight 
mattered  little.  More  important,  though,  for  the  purposes  of  this 
discussion  is  the  fact  that  both  studies  found  a  very  strong  relationship 
between  predictions  derived  from  the  statistical  model  (regression  model) 
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and  those  derived  from  subjects'  subjective  policy  statements.  Schmitt 
sums  up  the  findings  of  both  studies  when  he  says  "The  results  also 
suggest  that  weights  in  policy-capturing  studies  (e.g.,  Borman  and 
Dunnette,  1974)  would  not  need  to  be  statistical  weights.  The  rational 
judgments  of  cue  weights  are  equally  as  good  as  the  least-squares  weights 
derived  from  regressing  subjects'  judgments  on  the  cues  in  the  sense  of 
producing  predicted  values  that  correlate  highly  with  optimal  predic¬ 
tions  of  the  criterion  of  interest." 

An  additional  point  supporting  the  use  of  rank  weighting  approximations 
involves  the  problems  suggested  for  the  use  of  equal  weighting.  The 
correlations  between  attributes  found  in  Otway  and  Edwards  (1977)  and 
Newman  (1978)  raise  serious  doubts  as  to  the  stated  generality  of 
conditions  for  the  "equal  weights  theorem"  detailed  in  Wainer  (1976) 
and  expanded  in  Mainer  (1978).  Correlations  were  not  calculated  between 
attributes  for  the  LASD  analysis.  This  was  due  to  the  complexity  imposed 
by  the  hierarchical  MAUM  structure.  Values  at  any  level  of  the  hierarchy 
were  dependent  on  both  weights  and  values  from  lower  levels.  In  each  of 
the  other  two  analyses,  however,  several  dimensions  correlated  negatively 
in  excess  of  -.6.  As  previously  stated,  the  frequency  of  the  occurrence 
of  the  high  negative  correlations  in  the  two  studies  where  examination  was 
possible  suggest  that  Wainer's  conclusions  are  not  justified  in  view  of  the 
severe  and  common  violations  of  his  condition. 

In  addition  to  the  rank  ordering  of  attributes,  an  additional  bit 
of  information  is  required  for  two  of  the  rank  weighting  techniques. 

For  rank  exponent  and  decision  rule  rank  weights,  the  proportion  of  the 
total  weight  given  to  the  most  important  dimension  must  be  elicited  from 
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the  decision  maker.  For  the  purposes  of  these  re-analyses  it  was  assumed 
that  the  weight  given  to  the  first  dimension  by  ratio  weights  was  the 
correct  weight  for  that  dimension.  Clearly,  whether  subjects  can  provide 
estimates  of  this  quantity  is  an  empirical  question. 

Work  in  progress  will  seek  to  test  this  question.  Using  an 

external  criterion  of  utility  weighting,  this  work  will  test  rank 

weighting  procedures  against  both  the  established  "true"  weight  and 

the  weights  derived  from  subjects'  judgments  of  those  "true"  weights. 

If  in  fact  subjects  can  provide  estimates  of  the  most  important  dimension's 

weight  which  are  both  reliable  and  congruent  with  the  established 

"true"  weights,  the  results  of  the  current  study  indicate  that  a  simple 

decision  rule  strategy  for  selection  of  weight  approximation  procedure 
is  remarkably  effective  for  decision  purposes.  In  each  of  the  two 

analyses  where  one  level  of  attributes  was  involved,  the  simple  decision 
rule  procedure  for  selection  between  equal,  rank  sum,  rank  reciprocal 
and  ratio  weighting  led  to  selection  of  the  technique  which  correlated 
highest  with  the  results  of  ratio  weighting.  For  the  LASD  analysis,  the 
use  of  the  decision  rule  strategy  on  each  level  of  the  hierarchy  led  to 
a  remarkably  good  approximation  of  the  analysis  using  the  entire  set  of 
elicited  ratio  weights.  The  quality  of  this  approximation  (Pearson  r  =  .996) 
is  so  good  as  to  indicate  that  the  effort  involved  in  the  elicitation  of 
over  150  ratio  weights  was  unnecessary  and  the  decision  rule  rank  weighting 
procedure  a  more  than  adequate  approximation.  In  addition,  rank  exponent 
weights  seem  to  consistently  provide  estimates  of  the  ratio  weights  which 
have  near  the  same  level  of  peakedness.  Looking  at  the  regression  analyses 
between  the  set  of  ratio  weights  and  rank  exponent  weights  for  each 
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problem  we  found  that  slopes  were  generally  near  the  correct  1.0. 

This  finding  suggests  that  rank  exponent  weights  will  provide  good 
approximations  to  ratio  weights  in  most  instances. 

The  other  possibility  is  that  judges  will  not  be  able  to  reliably 
provide  the  weight  to  the  most  improtant  dimension.  In  this  case  we 
still  find  that  the  rank  sum  procedure  provides  an  approximation  that 
is  better  than  equal  weights.  In  sum,  we  have  a  situation  where  rank 
weighting  has  been  shown  to  be  superior  to  equal  weights  throughout 
the  reanalyses  conducted  in  this  paper.  This  is  true  whether  or  not 
the  more  stringent  assumptions  necessary  for  the  use  of  the  decision 
rule  and  the  rank  exponent  weighting  procedures  are  justified.  Certainly, 
the  moderate  increase  in  effort  necessary  for  the  elicitation  of  rank 
ordering  of  dimensions  is  justified. 
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